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To all whom it may concern: 



Be it known that We, 



Markus ANDREASSON and Per ASTRAND 



have invented certain new and useful improvements in 



METHOD AND DEVICE FOR RECORDING OF INFORMATION 



of which the following is a full, clear and exact description: 
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TECHNICAL FIELD OF THE INVENTION 

The present: invpnf.ion relates to a method and a 
device for recording of Le^L by iuiayixiy the Le-x.L un axx 
optical, sensor in a handheld device, said sensor being 
10 intended for digital recording of images . 

BACKGROUND OF THE INVENTION 

It is sometimes desired to abstract parts of text or 
iuiaye ixxIcruiaLiuxx in a ducuiuexxL which may later be edited 
flj 15 using appropriate software in a computer. A known method 

ul ixxputtixxy Le-xL axxd image ixxrurmaLioxx ixito a computer 
is to use a stationary or portable scanner. 
M: A stationary scanner is suitable for entering entire 

M : pages with text and image information, the scanner being 

!7i 20 automatically passed aciObb the paye aL a cuxxataxxL ypeed. 

X This type of scanner is not suited for inputting selected 

M : parts ol information on a page, 

A portable, handheld scanner may be used any time 
interesting inlormatioxa is Lo be scanned, but normally 
25 has a limited field of view. 

US-5, 301,243 disclose* a lxaxxd-lxeld acaixxxei. lux read- 
ing characters from a string of characters on a sub- 
strate. The scanner is moved in contact with Lhe 
substrate along the character line and has an optical 
30 system which images a small part of the substrate. The 

optical system comprises a CCD type line sensor provided 
with a plurality of liqht- sensitive elements arranged In 
a line. When the scanner is passed across the characters 
on the substrate, a succession of vertical slices of the 
35 characters and of the spaces between them is recorded. 

The slices are stored in the scanner as a digital bit-map 
image. OCR software (OCR =* Optical Character Recognition) 
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is t-.hftn iiRcarj t.n identify the* nharanhars anti shorfi hhPTn in 
character-coded form/ such as ASCII code. The character 
TRnngrii t.i on ran by madp. Rithp.r in thfi Hn^rmp.r nr in an 
external computer to which the bit-map image is sent. 
5 Another type of hand-held scanner tor inputting text 

is disclosed in US-4, 949, 391. This scanner has a two- 
dimensional flftrifior whir.h rfir.ords images of the underlying 
surface as the scanner is being moved across the same. 
The scanner is restricted to movements in a direction 
10 which is determined by a wheel in contact with the 

surface. Before the recorded images are assembled into a 
composite image, redundant information is removed from 
"*wjP the images. The composite image can be analyzed in a 

f% i 

computer for identification of characters. 



15 A drawback with the handheld scanners described 

dbovti la LhaL Llieii. field of view its xelaLively small. In 
order to record a large amount of information, such as 

pa 

y : passages consisting of several lines, a user must 

yj therefore move the scanner back and forth across the 

Cl 20 surface repeatedly. Moreover, the movement has to follow 

a predetermined path, such as along the lines of text. 

Publication WO 99/57678 discloses a device fox- 
recording information from a substrate. The device may 
operate in two modes, one scanner mode, in which lines ol 
25 text is scanned, and a photograph mode, in which separate 
pictures are taken of a document or an object. 

Publication WO 98/20446 discloses a scanner pen, 
which is adapted to be moved over a line of text for 
scanning the text. As the pen moves over the text image, 
30 several pictures are Laken of Lhe LexL . The piuLuiea axe 
processed by a computer and assembled or stitched 
together for forminq a composite image of the entire line 
of text, which cannot be read by a single picture. The 
seamier yen can only scan a sinyle line of Le.x.L at each 
35 t ime . 

Thus, there is a need for a handheld scanner pen of 
the above-mentioned type which is adapted to scan several 
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Iitirh of tp.vt* RiTTinlt^TiPoi.i8ly as well as smaller pictures. 

SUMMARY OP THE INVENTION 

An object of the present invention is to provide 
.*> a RHranriRr ppn whinh Rnahl ft faRt "recording of text in real 
time. 

Another objpnt of the invention is to provide a 
scanner pen which may be used &L a diaLeuiue £ium the Le*L 
and may scan several lines of text in a single stroke as 
10 well as discrete pictures, 
p These objects are obtained by a method and a device 

Q fox' xecurdiiiy iiilujiuiciLiuii by imayiiiy on a liyhL-seusi tive 

*-JP sensor for obtaining at least two images o£ the 

In 

>; infoiiudLiuu hdviuy partially uverlappiny cunLenLa . The 

1 S method comprises converting the information in each of 

Lh« ituayea Lu cuded j.eprteifc>eiiLdLiuii, cuiupariny the coded 
representation of said images for determining an overlap 
posiLiuju, <diid dsseuiblyixiH Lhe iiuaqtifci Lu form d uumpoyiLe 
image. The coded representation may be a character code, 
20 such as ASCII. Alternatively, the coded representation 
pi may comprise a division of the information inside 

budrdera, uuch a* recLanyleu, eduli comprisijxy portions of 
the information, such as words included in said 
information. Thereafter, the composite imaqe may be 
25 converted to a character code format, such as ASCII-code. 
Alternatively , each imaye may be separetly converted into 
character code format, such as ASCII, before assemblying. 

The method may further comprise determininq 
structures in each of said images, such as direction of 
30 lines or text line directions in each imaqe. This may be 
accomplished by means of a Hough transformation of each 
imaqe. This information may be used for adjusting the 
rotational position and/or perspective of each image in 
dependence of the direction of lines. The information may 
35 also be used for the division of the image in 
reaetaiiyletp . 

A concept of the present invention is to record a 
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plur-sl itvy of imagpR of r tevt , said images overlapping 
each other, each image comprising several lines of text, 
finbR«rp.ient1y r OCR (Optical Character Recognition) is 
carried out as regards the recorded images, resulting in 
3 Sfif.fl of nharantfirfi. ThR RRtR of nh^racters may contain a 
number of characters indiceitiiiy "end u£ line" if the text 
comprises several lines. Then the sets of characters are 
assembled lining the dididuLeia lu the aety of characters. 
An advantage of this mode of operation is that relatively 
^1% 10 large images can be recorded at a time, without the 

Q assembling of the images being cumbersome since the 

y images are converted into character cuJwb befuxe being 

Ifi 

assembled. The effective resolution is small in a set of 
pj characters compared Lu a bit-map intake, thus saving 

Hf 15 computing power. The effective resolution in the set of 

characters is a single character. Thus, assembling in two 
y : dimensions may be possible with the present invention in 

H a handheld device. 

The lines of text in two subsequent images do not 
20 necessarily coincide in the vertical direction. Thts first 
line in a first recorded image may correspond to the 
second line in a second recorded image. However, the 
assembling will adjust the vertical position so that 
correct assembling is obtained. 
25 By a set of characters is meant a plurality of 

characters, from which the relative positions of the 
characters can be determined. The set of characters may 
be a strinq of characters comprising characters lor blank 
and end of line, 
30 A device for recordinq a text imaqes the text; on a 

light-sensitive sensor with a two-dimensional sensor 
surface, which sensor is intended for diqital recordinq 
of images of the text, said images having partly 
overlapping contents. The device is characterized in that 
35 it is adapted to convert at least two of the images 

xecuxded by the seusux into a set uC characters each, 
comprising a plurality of characters, by means of 
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character recognition. The device is adapted to 
subsequently assemble the sets of characters with the aid 
of the characters in the sets of characters. 

By carrying out character recognition before 
5 ARRRTnhl ing t.hft im^gRR, f.ho opp-r*t.i on of aRflfimhl ing l^rge 
images aL pi^el level can be omitted. Moreover/ there is 
less risk that a character would not be recognised owing 
Lu poor assembling , which may be the udati when digital 
images are assembled in pixel level and then character 
10 recognition is carried out in an area that is overlapped 
by both images that are assembled. Then, the character 
may be distorted if the assembling is not carried out 

\Jt\ properly, which may results in that the character is not 

■£ 

^ : xecuyni^ed in Lhti chdiduLtii xecoyniLioii p-LOces* . 

m 15 According to this invention, the character recognition 

*P process cake advantage, of the original quality of the 

image for character recognition- By first converting the 

Mi 

L, iuianeu iiiLu sets ol uhdiduLeib, Lhe acLual assembliim u£ 

the sets of characters may be quick since the number of 
20 chdiduLeib it. considerably smaller than Ihe number of 
y' pixels in the recorded images . 

Moreover, by OCR interpretation ol each image before 
the assembly thereof, a plurality of OCR interpretations 
of the same character will be obtained, one for each 
25 image where the character is included, and the 
iiiLexpreLaLioii which qivea hiMhesL xecuqiiiLiuii 
probability can be selected. 

Alternatively/ it is possible Lu asseiiible the sets 
of characters using words in the set of characters. Thus, 
30 entire words in one of the sets of characters that are to 
be assembled are compared with words in the other of the 
sets of characters that are to be assembled, when 

assembling words, it may be required to compare each 
individual character. 
35 By word is meant a plurality of characters which 

also includes special characters. The special characters 
are, for example, blank, full stop, comma or end'of line. 



y 
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ThR method may r.nmpri rp. finding thp. linpp of f.«vt in 
the recorded images, to find start and end of words along 
the Lines of text and to select which of the recorded 
images are to be converted into sets of characters with 
.*> the* aid of the* identified Atari", and And of the* words in 
the recorded images , so that only images with the 
nRf i ftR ^ry information srfi nnnvprtP.H into set* of 
character a oi LhaL images with duplicate information may 
be discarded. By identifying start and end of the words 
10 along the lines of text, it will be possible to make a 
rough assembling of the images without first making 
optical character recognition. When the start and end of 
^JJJ the words have been identified, the recorded images are 

correcLed as regards rotation and perspective. By makiuy 
15 a rough assembling of the images, it will be possible to 
find out how the iuiayeu overlap each other. The imayes 
5 " which contain only information that is available 

M" completely in other images then need not be converted 

p into sets of characters . 

hj 20 SLarL and end of words along the lines of text may 

be identified by searching, in each pixel along a line 
thruuyh the line* of LexL, lui the number of ddik pixels* 
a predetermined number of pixels up and down from the 
line of text. An exid o£ a word being delined as il there 
25 are no dark pixels in a predetermined number of pixels 

abuve and below the line of text, i.e. there are blaxika . 
To manage italics one may alternatively search along an 
oblique line. IL ia, of course, possible Lu searuh fur 
white pixels, instead of dark pixels, if the text should 
30 be brighter than the backyruuud, i.e. inverted. 

The images may be converted into binary images, i.e. 
imaqes containing merely black arid white, since this 
facilitates the continued image processing . 

Moreover, the method may comprise finding the lines 
35 of text in the recorded images using the Hough 

transformation of the recorded images. The Hough 
transformation can briefly be described as follows. There 
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is an infinite number of straight lines extending through 
a point in an XY plane. The equation of the straight line 
for gach of these lines can be expressed with two 
parameters. If the parameters of Lhe individual lines are 
5 pi nt-.t-.prf in a diagram, r curve i s obtained which 

corresponds to the Hough transform of Lhe puinL. In Lhit* 
way, it is possible to plot curves for any of the points 
in Lhe XY plane. The Hough Lransform of two different 
points will intersect in a point, which corresponds to 

10 the equaLiun of Lhe bLraiyhL line e*.Lendiny Lhrouyh Lhe 
two points. If the Hough transforms for all the dark 
pixels in a xuuuidud iuieLye axe pluLLed, a larye number of 
intersections between the different Hough transforms will 
be ubLaixxed, However, there is a maximum nutriber of 

15 intersections for lines following the lines of text. 

The device may comprise a memory adapted no snore 
the recorded images, which are to be converted into sets 
uf uharticLery, in Lhe memory , and Lo convert Lhe sLored 
images into sets of characters after completion of the 

20 recording of the images. By only storing the recorded 

images which are to be converted into sets of characters, 
Lhe memory space which is required for storing recorded 
images is minimized. By converting the stored images into 
sets of characters after the recording of imaqes has been 

25 completed, it is not necessary to place high demands on 
Lhe speed of Lhe opLical characLer recognition, which 
would be the case if it were to be carried out while 
images are being recox'ded. 



30 ner that a user can hold it by hand and at a distance 
from a substrate to record text on the substrate „• 

The device may be adapted to correct the images for 
rotation before they are converted into sets of 
characters. In the case where the lines of text in the 

35 recorded images have been identified, the correction for 
rotation can be carried out in a relatively simple way. 
However, some optical character recognition programs can 



The device is advantageously designed in such a man- 
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process also rotated images, in which case the rotation 
is not required. 

florrespondi ngly, the device may correct the images 
for perspective before they are converted into sets of 
5 characters since the optical character recognition may hp. 
facilitated if all letters have the same size in the 
images . 

The device may be designed as a reading head which 
is connecter! to a calculating unit in which the records 
' w 10 images are processed. 

!Z The (ipvi hr may bp. artflptp.ri to assemble the Rets of 

characters by comparing the sets of characters in pairs, 
\.Lfj the sets ot characters being compared in a number of 

relative positions displaced relative to each other. 
\5 Thus, a first character in a first set of characters ir 
compared with characters in the second set of characters 
until correspondence is achieved or until thp. first 
character in the first set of characters has been 
y : compared with all characters in the second set of char- 

20 acters. Subsequently, the correspondence of the second 
characters in the first set of characters is compared 
with the characters in the second set of characters. By 
making the comparison for a large number of different 
relative positions, a plurality of total numbers of 
V..*> points can he obtained, thp. total number of points 

reflecting the correspondence between the two sets of 
characters for the specific position. In this way, an 
optimum x-elative position can be obtained. 

The device may be adapted to store the recorded 
30 images that are to be converted into sets of characters 
along with a serial number indicating in whinh order the 
images have been recorded, and to assemble the sets of 
characters w.ith the aid of the ^p-Hal number for the 
images corresponding to the sets of characters. 
At) Especially in the case where first all images are record- 
ed and not until then the character recognition and the 
assembling are begun, the serial numbers may he used for 
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the recorded images since then a large number ot images 

are to be assembled. '[ 

According to a second aspect ot the present inven- 
tion, a method is provided for recording of text on a 
5 substrate, comprising the seep of imaging and digitally 
recording images of the text, the images having partly 
overlapping contents. The method is characterized in that ; 
it comprises the steps of converting at least two of the 
recorded images into a set ot characters, each with a j 

10 plurality of characters, by means of optical character 1 
recognition, and putting together the sets ot characters 
with the aid of the characters in the sets of characters . 

The a -rea recorded by thR RP.nRnr may arranged sn ^ 
that a plurality of lines of text are imaged in a 

1!* recorded image. 

According to a third aspect of the present inven- 
tion, a computer-readab.l e storage medium, in which a 
computer program is stored which is adapted to be used 
for nnnvRrsinn of digital images, which are retr.nrcieici by 

20 an image sensor , into text. The storage medium is char- 
acterized in that the computer program comprises Instruc- 
tions for making the computer receive digital images as 
input signal*, convert the digital image* into setps of 
characters, with a plurality of characters, by means of 

2S nharantRr rfinngnitinn, and put together the flfthfl .of char- 
acters with the aid of the characters in the sets of 

Further objects, features and advantages of the 
invention will appear from the following detailed 
30 description of embodiments of the invention with 
rpfpTftnce tn th«=> drawing*. 

ftPTFlF nKSfTRTPTTON OP TTTF! DPAWTNOS 

Fig. 1 is a schematic view of a device according to 
35 a tirat embodiment ot the present invention. 

Fig. 2 is a schematic block diagram of parts of the 
embodiment of Fig 1. 
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Pig. 3 is a schematic view of images of tevt on a 
sheet of paper, which are recorded according to the 
invention. 

Figs, 4a and 4b are diagrams, illustrating the 
h principle of the* Hough transformation . 

Fig- 5 is a diagram, which shows maximum points for 
the Hough transform nf twn rH f f p.rfint -imagRS. 

Fry. 6 iu d diayram and a histogram for illustrating 
the detection of start and end points of words. 
— * 10 Figs. 7a and 7b are diagrams for illustrating the 

H' division of the text images into words, 

y Fiy. 8 is a diagram of two images converted to 

^fl characters for assembling. 

=p Fig- 9 is a diagram for illustrating lines of text 

JJj 15 in a recorded image. 

% Fig. 10 is a, How chart u£ the operation u£ a cum- 

s puter program according to the invention . 

H : F i q . iid, nb and 11c are shemaLic representations 

of a text, division thereof in rectangles and display on 
20 a small display. 



^ DETAILED DESCRIPTION OF EMBODIMENTS OF THE INVENTION 

Fig. 1 discloses a scanner pen, comprises a casing 1 
havinq approximately the same shape as a conventional 
25 highlighter. In one short side of the casing there is an 
openiuy 2, which is intended to be directed at an area un 
a substrate which a user wants to image. The information 
udixiw can be a sheet of paper. 

The casing 1 essentially contains an optics part 3, 
30 an electronic circuitry part 4 and a power supply part 5, 
The optics part 3 comprises a lens system 7, light - 
emitting diodes 6, and an optical sensor 6 constituting 
an interface with the electronic circuitry part 4 . The 
light -emitting diodes 6 may be used to increase the 
35 illumination . 

The light -sensitive optical sensor 8 may be a two- 
dimensional CMOS unit or CCD unit (CCD = Phar-gR Cnupl rH 
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Device) with a built-in AD converter. Such sensors are 
commercially available- The sensor 8 may be mounted on a 
printed circuit board 11. 

The power supply to the device is obtained from a 
5 hatt.p.ry 12 which ir mmmtfiH in s ffeparate compartment 13 
in the casing. 

Fig. 2 is a V>1 nek Rcbema of the electronic circuitry 
part 4, which comprises a processor 20 which, via a bua 
21 , is connected to a read-only memory ROM 22, in which 
10 the program of the processor is stored, to a write/read 
memory RAM 23, which is the work memory of the processor 
and in which Lhe images from Lhe sender aa well at> 
characters that are interpreted from the recorded images 
are sLured, Lu a conLrol logic unit 24 and Lu Lhe sensor 
IS 8. 

The conLrul luyic uuiL 24 is cuuaecLed to a number 
of peripherals, such as a display 25 mounted in the 
casing, an IR transceiver or short-ranqe radio link 26 
for transferring information to/from an external . 
20 . cuiupuLex', buLLuns 27 by means ul which Lhe user can 

control the device, and an operation indicating device 28 
cuiisisLiny ul a second seL el liyhL-eiuiLLixiy diodes which 
may indicate whether recording occurs or not and other 
uperaLiuy condi Lions , The cuuLrul luyic uniL 24 yenerates 
25 control signals to the memories, the sensor and the 
peripherals. The control loqic uniL also uianaqes Lhe 
generation and the prioritization of interrupts to the 
processor. The buLLous 27, Lhe Lransceiver 26, the 
display 25 and the light -emitting diodes 6 are controlled 
30 by Lhe processor by wriLiny and readiny in Lhe records of 
the control logic unit. The buttons 27 generate 
inLerrupLs Lo Lhe processor 2 0 when activated. 

The function of the device will now be described, A 
sheet of paper 9 is provided with a plurality of .lines of 
35 printed text 10 as shown in Fig. 3. When a user activates 
the scanner pen by means of the buttons 27 and passes iL 
across the sheet of paper with the opening 2 directed 
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towards the sheet of paper, three images 14, lb, 16 are 
recorded. Each of the imagco 14, 15, 16 io cxpoocd to OCR 
prorres sing and the text ot the images is converted into 
set3 of characters as illustrated in Pig. 0. Gubaequently 
5 the sets ot characters are assembled or stitched so as to 
form a complete text- As shown in Fig. 3, the images 14, 
15, 16 may be rotated in relation to each other- Thus, 
a first image 14 is turned or rotated in relation to a 
second image 15, which in turn is rotated in relation to 
^ 10 a third image 16. 

q Tn nrHfir to optimize*, the optical character recogni- 

Q tion in the recorded images 14, 15, 1C it is advantageous 

.tj to know the orientation ot thp Iitip.r nf t.pxt in thp 

image. Therefore, the orientation of the lines of text is 
(rj i$ dpf.pr.tpd hRforft the character recognition is carried out. 

4* The detection of the orientation of the lines of 

J\ tp.vt may bp narrifiri out using thp TTnugh transformation. 

: U Referring to Figs 4 and 5, the Hough transformation 

M* will now bp gpnp.rally dPRnribpd. Pig. 4a fibowfl fivfi 

20 points in a plane with the coordinate axes X and Y. Pig 
4b shows the Hough transform ot the five points in Pig. 
4a. A first point 18a has a first Hough transform curve 
19s which dpsnrihRR all thp straight linps pxtRnding 
through the first point 18a in Fig- 4a as a function of 
7.3 thp paramptprs H and p whprp ft i s thp angl « of the 

straight lines through the point and p is the distance of 
the straight lines from origin. The Hough transform 
curves have a sinusoidal shape. Correspondingly, the 
second 18b, third 18c, fourth 19d and fifth 18e points 
30 have <x second 19b, third 19c, fourth 19d and fifth I9e 
Hough transform nurvp. Thp Rpnonri 19h, thi-rd 1 9r and 
fourlh 19d Hough transform curves in Fig. 4b intersect in 
a point. This point in Fig. 4b r.nr-rwpnncla to a straight 
line in Fig- 4a extending through both the second point 
IRb and thp third point 1 8r. and the fourth point 18cL 

Fig, 9 shows the second recorded image 15 from 
Pig. 1 , which confiifits of a plurality of pixels in which 



a 



'01 11/13 TIS 10:27 FAX +46 40 260516 



AWAPATENT AB 



10 017 



13 

lines of text arft to be tound. For each black pixel in 
the 3econd recorded image 15, a Hough transform curve is 
calnnlated. Th« Hough transt orms are inserted in one and 
the same diagram. 

T?ig. 9 shows a tirst line 32 and a second line 33. 
Lines having approximately the same direction as the 
first linp. 12 wi I I intersect a larger number ot points 
than lines having approximately the same inclination as 
the second linp. 11 si nop the dark pixels in the recorded 
^ 10 image 1G are positioned along lines of text having 

M= approximately the same direction as thp first line 12. 

« Ficr. 5 illustrates that the maximum number of 

vJJ; intersections can be used to determine the orientation 

£ of the lines of text. The circles 73 correspond to points 

Hsj 15 in the Hough tranRform diagram whp.rp Rfivprsl Hough 

J; Liautjluxui uuxves of points in Fig. 9 intersect, i.e. 

maximum of intersections. Thp nirnlpR 7^ corresponds to 
M= the image 15 and are positioned along a straight line 34. 

:~ Prom the distance between the circles 73, the rH nt-.*nce 

j 5 ; 20 between the lines of text can be determined. The position 

Q of the intersecting line 34 along the 0 axis indicates 

the xuLciLiun of the recorded image. The second line 35 in 
Fig. 5 corresponds to the imagp 14 and pytpnris through a 
pluxdliLy u£ itiemiiuum indicated by crosses 36 in the 
25 diagram. The inclination of the second line 35 indicates 
that the image has a perspective, i.e. the lines of text 
have different rotations. Also the sligth different 
distances between the crosses 3 6 indicate that the image 
has a perspective with larger distances between the lines 
30 dL Llie luwei puxLion of the line 36, The displacement of 
the second intersecting line 35 in relation to thp first 
line alonq Lhe 6 axis indicate* LliaL Lhw lines of texL 
are rotated in the recorded image. By means of this 
Inform* Lion, Lhe iuiaye may be adjusted for perspective 
IS and rotated, for pxampl r to the* horizontal direction, 
which is the same direction as image 15. As appears, 
image 15 corresponding to linp. 14 n^^rj no adjustment. 
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while image. 14 corresponding to line 35 needs adjustment 
of the perspective to make line 3 6 vertical and with 
approximate! y equidistant crosses 36 and adjustment as to 
the rotational position, to move line 3 5 to the same 
5 angle 9 as image lb, which may correspond to zero angle. 

After identification of the lines of text, an iden- 
tification ot the start and end parts of the words in the 
recorded image is carried out- Pig- 6 indicates how the 
iRtt-.pr "p." ^7 is detected. As shown in Fig. 6, the number 
* 10 of dark pixels 7 are counted in the vertical direction 

Jf; perpendicular to the line 3fi, which may he calculated as 

H described above. The number of dark pixels is zero up to 

<Jrj the start 353 ot thR 'I fitf.Rr "e 11 and will again be zero at 

t: the end 4 0 of the letter "e" . When the number of dark 

m 15 pixels has hflfin 7P.ro for a predetermined period, this is 

=P detected as the end of a word. 

f With rftffirfinnp. to Pig. 7, the words are indicated as 

L rectangles, the start 41 of the rectangles indicating the 

M start of a word and the find 42 of thfi rectangles 

20 indicating the end of a word. Fig. 7a corresponds to a 
first recorded image U and Fig. 7h norrfiflpnnds to a 
second recorded image 15. A length of a first word 43 
in Fig. 7a has correspondence in a length ot a second 
word 44 in Fig. 7b. Correspondingly/ a length of a third 
75 word 45 in Fig. 7a has correspondence in a length of a 

fourth word 4 6 in Fig, 7b, By matching the two images, it 
is possible to find out how the rp.nnrrlprt images over! ap 
each other, by only using the graphical information of 
the length of each word. 
30 Thus, by using the length of the words for each 

line, it is possible to ca-rry out a rough putting- 
LuyttLliex.- ui aLiLching or assembling of the two images. 
The images are roughly assembled sn that a serjnenoe of 
word lengths in the first recorded image corresponds to a 
'■W sequence of word lengths .in the second recorded image. 
The word lengths along different lines in the first 
recorded image should thus correspond to word lengths 
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along corresponding lines in the second recorded image* 
In this way, it is possible to determine how the imagoo 
arft diffp"lan«ni relative to each other. 

Pig, 3 shows how this may be used to sort out 
5 unnecessary images so that: optical character recognition 
need not be carried out as regards all imaged that arc 
recorded. A first image 47 and a second image 49 com- 
pletely overlap the area which is covered by a third 
renorrip.fi image 48 which is indicated by dashed lines. By 
^ 10 using the method as described in connection with Fig, 7, 

y the third r^nnrdp.d image 4fi can he completely omitted, 

p without optical character recognition being carried out. 

^0 0t course, Fig. 1 i« only schematic-!, since the images 

*L : j often overlap each other to a much larger extent. Indeed, 

S] 15 it the scanner pen is held approximately still, all 

CD images overlap more or less, and a substantial saving of 

+* computing pnwp.r may he ohtained by omitting images with 

y= duplicate information. On the other hand, during normal 

scanning operation, a partial overlap may hp. user! for 
N : 20 sorting out errors in the optical character recognition 

~; prnnpRg or thp aRRomhling process, in which case 

iU overlapping images are not omitted or discarded. 

Another method ot assembling the images is disclosed 
in Fig. 8/ which shows a first set of characters 50 with 
75 a plurality of charartprs 60 corresponding to the first 
recorded image 47 in Pig. 3 and a second set of charac- 
ters 51 with a plurality of nharanhp.rR 61 corresponrii ng 
to the second recorded image 49 in Fig. 3. The text in 
the first set of characters and th« Rpnnnii set of char- 
30 acters follows the direction of the lines of text 62, 
which may have heen ohtained by the Hough transform 
process described above. The first set of characters 50 
and the second s«t of characters SI are put together or 
assembled by comparing characters in the two sets of 
AS characters. Firstly, the tirst character 52 in the tirst 
set of characters is compared with each of the characters 
in the second set ot characters 51. The operation 
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proceeds correspondingly for the second character 63 and 
the third character 64 in the first set of characters SO. 
uood correspondence is obtained when the characters in 
the word "pkillcd" 53 in the second line of the first set 
5 of characters is compared with the word "skilled" 54 in 
the first line in the second 3Ct of charactcra 51. Since 
a word can appear in many positions in a text, one starts 
from the first correspondence found and then comparco the 
rest of the text for this position, a total number of 
10 pointc being obtained which indicates how well the two 
p sets of characters correspond for this position. 

G Subsequently this step is repeated for the next pooition 

"Oil where correspondence is obtained. Finally, the position 

nj is selected in which the total number of point3 indicates 

BP 15 the best correspondence- in Fig. 8, the text is in 

*? : English but a person skilled in the art under otando that 

y the text could just as well be in any other language and 

M: that the text in Pig. 9 ic only uced to illustrate the 

H ! function of the device. The image may also comprise 

Is : 

JSj 20 symbols, such as mathematical expressions, or even line 

M: art, to a limited extent. 

The two assembling methods may be combined, so that 
the images are first compared in a rough manner and then 
* w on character level. In this way, it will be possible to 

25 carry out assmebling in two dimensions with limited 

computing power. If the reading device ic firct moved to 
^ the right like in Fig. 3 and then down and to the left, a 

larger surface is obtained which is to be assembled in 
two riimp.rifii oris . Ry determining the mutual relationship ot 
30 the partial images by rough assembling and then on 

character level, it is relatively easy to obtain both 
horizontal and vertical assembling. The charactcriotic 
that the lines determine the vertical position with great 
accuracy results in the possibility of alignment. 
35 Each assembling method may alternatively be used 

separately. If the method ueing length of words is used 
separately, the images may be assembled into a composite 
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image* before final conversion of the images co characters 
by optical character recognition, Thio method may be uccd 
wh«n th« images comprises other matter than text, such as 
small pictures or symbols. In this case, the pymbolo or 
5 small pictures will be handled as if they were words and 
be included in a rectangle. The assembling process will 
work, provided that the pictures or symbols are 
sufficiently small- However, the OCR process will fail to 
T«nngrni7.p f.h« rectangle as characters, and then the pixel 
10 representation may be used instead. Other types of 

hnarHfira may used than rectangles, such as an area 
delimited by more than four straight lines or even curved 
lines . 

If the assembling is carried out after the 
15 conversion of ftanh imagft to character code such as ASCII , 
small pictures and symbols may be handled separately, at 
s least if they are surrnnndpri by t.mrh matter in each 

image - 

As appears from Fig. 7 , hhft imagp.fi may be processes 
hj 20 by division of the image in boarders, such as rectangles. 

C! Each rectangle comprises a complete word or a symbol or a 

p " : picture surrounded by white areas. Several pictures are 

compared for finding a snnnpflRinn nf rfirtsng'l ft?; which 
^ cuirtjapuiids Lo Lhe wanted text, When overlapping 

25 positions have been dp.tevrmiripd, p.anh rfintang'l p. is givan a 
succession number, comprising the line number and the 
word number on that line. ThnR, in Pigs. 7a and 7R, th<= 
word 41 obtains designation 1:1, the next word 1:2, and 
the following word 1:3. On the second line, the first 
30 wuik ubLcLixiti Lhe designation 2:1, the second word 2:2, 
the third word 2:3 and the fourth word 2:4- 
Curxeypuujiiiyly , un Lhe third line, Lhe first word 
obtains designation ? : 1 , the spnnnd word ~*z?. r the*, third 
word 3:3 and the foux"th word 3:4. It is now recognized 
■■$.*> that thp fourth word on the second row, 2:4 corresponds 

to the second word on line 2 of Pig. 7b, which means that 
thfi first wnrd at Mift second line of Fig_ 7b obtains 
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designation 2:3, the second word 2:4 (corresponding 10 
the fourth word of line 2 of Pig. 7a), the third word 2:5 
and the fourth word 2:6. The same goes for the third line 
of Pig. 7b/ in which the firot word obtaino designation 
5 3:3, the second word 3:4, the third word 3:5 and the 

fourth word 3:6. The words arc then arranged in a row of 
words forming a complete line. 

It can now be seen that several words arc duplicated 
in the two pictures, namely word 2:4 and word 3:4. There 
10 arc further words which arc duplicated on the other 

lines. These duplications may be omitted and replaced by 
a single word. Else the duplicationo may be kept and uood 
for increasing the OCR interpretation accuracy. 

Finally, the words arc OCR processed in the right 
15 order to obtain the desired text. 

If any rectangle is larger in the vertical direction 
than a single line, it may obtain designation like 2,3:6, 
if it occupies lines two and three. In thio way, larger 
objects such as pictures or symbols may be handled. 
20 There are a number of cases in which partial wordo 

are included in Kig. va but included in full in Fig. 7b, 
such as word 1:3- In this case, the longest version of 
the word is used for interpretation, if there is a doubt 
if this is correct, all fragments may be used to recreate 
25 the complete word. 

In this way, the images arc assembled on a word 
basis charting from the pixel representation and dividing 
the image inside boarders, such as rectangles which are 
comparer* for the best overlapping position. Then, 
30 duplicate information is omitted or used for further 
accuracy and then the words are arranged in the right 
order and finally converted to ASCII-code. 

Pinl-.urfcfl or symbols which may not be recognised by 
the OCR program, may be maintained in pixel format and 
35 displayed as such in the final picture, Jr'or increased 
oafcty, aloo at leant one version of the word© may be 
kept in pixel format in parallel with the OCR- version , 
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HRp«nially if thp. OCTR program lnriinahRfi a poor quality of 
processing. 

Every new image "is compared with previous 
information in order to find out its orientation therein. 
5 Thus, fiar.h image is processed both in the verticai 

direction as well as in the horizontal direction, both 
forwards and backwards. Thus, it is possible to scan in 
two dimensions by the present invention. This is possible 
because of the division of the image in a coded 
10 representation which is less time consuming to process, 
either in the form of rectangles or similar or in the 
form of ASCII code, 
vjj; It may be ot interest to have some kind ot teed-back 

«p that the desired information is gathered. This may be 

15 annnmpl i shpd by displaying thp asRP.mblp.d information at 
the soieeii. However, since the display at a handheld 
device is rather small , another method would be to 
display the lines as a succession of pixels at the 
display, in which one pixel approximately corresponds to 
Ly 20 a single character. Then, the characters will form words 

and the layout, of words would give a visual indication of 
the scanned surface. If the processing comprises division 
into rectangleR, thPRR rp.ntangl pr may hR indinat.pd as 
Lhey cuLti assembled. , 
75 Figs 11a, lib and 11c show how this may be 

accomplished. Fig. 11a is a text that is to be scanned. 
Pig. lib is the division of this text in rectangles. 
Finally, Fig. 11c is the representation of the rectangles 
at a small display, in which rectangles are indicated in 
30 black pixels and spaces between the rectangles as gray 
pivels. From Fig. 11 r it nan bp sppn that aomp 
infoxiuaLiuii is missing as indicated by white pixels 29. 
The user then directs his scanner pen towards the area 
missing until the display indicates that all areas are 
fully nnvprpd by at lpast onp. imagp. Finally, the* com! ptp 
image is converted to ASCII, if that has not been done 
parlipr in thp procpss. 



01 11/13 TIS 10:31 FAX +46 40 260516 AWAPATENT AB ^024 

20 

if che assembling is done by using the coded 
representation in the nature of ASCII code, each decoded 
character is displayed as a black dot an the display 
screen, while spaces are displayed aa grey dots - Any 
5 white dot will indicate that information is missing, as 
described above - 

According to the present invention, it is required 
that the information at least partially is positioned 
along identifiable lines, if the device is passed across 
^ 10 a photograph or some other surface which ia not divided 

m into lines, this can easily be recorded by the processor 

Q in the device, and this part of the image may be 

di flc^T-cip.d or stored separately as an picture or 
photograph. If said surface is completely or at least 
fU 15 partially surrounded by lines, it would be possible to 

handle the situation via the invention, as soon as at 
least a portion of a line is included in every image. 

£ 

u. Pig. 10 ia a flow chart of the operation of a com 

puter program according to the invention. The computer 
20 program is adapted to be executed in the electronic cir- 
g mii try part 4 of the device, in a first step hh, digital 

images are received from the sensor 0 . In a second step 
56, the digital images are converted into strings of 
characters using character recognition. In a third step 
25 57, the strings ot characters are assembled or put 
together . 

Tt is not necessary to carry out the rough putting- 
together of the recorded images as described in 
nonnp.ntion with Pig. 7, and optical character recognition 
30 can be carried out directly as regards the recorded 

images. Moreover, the orientation of the lines of text 
need not be identified in the recorded images if an 
optical character recognition algorithm is used which is 
able to identify characters also when the lines of text 
35 are rotated. 

A person alcillcd in tho art rcalizco that the inven- 
tion is not limited to the embodiments shown and that 
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many modi fi cat i nns are f>.afiibl« within tbp .<=inop« of the 
invention. The invention is only limited by the appended 
patent rlsinis. 
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